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ABSTRACT 




STUDENTS, DIRECT 


USING ACADEMIC ACHIEVEMENT DATA FOE 655 UNIVERSITY 
VALIDATION OF DIFFERENTIAL PREDICTIONS BASED ON A 



BATTERY OF AFTITUDE/ACHIEVEMENT MEASURES SELECTED FOR THEIR 
DIFFERENTIAL PREDICIIOM EFFICIENCY WAS ATTEMPTED. IN THE 
CROSS-VALIEATICN OF THE PREDICTICN OF ACTUAL DIFFERENCES AMONG FIVE 
ACADEMIC AREA GPA*S, THIS SET OF DIFFERENTIAL PREDICTORS HAD ONLY A 
SLIGHT ADVANTAGE OVER A SET SELECTED TO MAXIMIZE ABSOLUTE PREDICTION 
OF THE FIVE AREA GPA'S. THREE WAYS IN WHICH THE EFFECTIVENESS OF 
DIFFERENTIAL PREDICTICN CAN EE IMPROVED ARE PRESENTED. (AUTHOR) 
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Direct Validation of Differential 
Prediction^ 

Clifford jS. Lunneborg 



Using academic achievement data for 63^ university students 
direct validation of differential predictions based on a battery of 
aptitude/achievement measures selected for their differential 
prediction efficiency was attempted. In the cross-validation of the 
prediction of actual differences among five academic area GIVl's this 
set of differential predictors had only a slight advantage over a 
set selected to maximize absolute prediction of the five area GBl's. 
Three ways in tdiich the effectiveness of differential prediction 
be iaqproved are presented. 
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Direct Validation of Differential Prediction 
Clifford iil. Lunneborg 

In a pair of monographs authored in the middle fifties^ Horst 
(195^; 1955) drew an important distinction between two possible goals 
associated with the selection of predictor variables where success in a 
nuniber of criterion activities is to be predicted. On the one hAnd pre- 
dictors may be selected which together will have the greatest predictive 
efficiency over all of the criterion activities. As usually Interpreted 
this means selecting a set of predictors, of a given size, which will 
yield the highest average multiple correlatioii over the several criteria, 
and Horst ( 195 ^) bas referred to such a selection strategy as mAyifiriy 4 Tig 
multiple absolute prediction. Qy contrast, a set of predictors, again of 
some given size, may be selected so as to maximize the efficiency with 
which the battery on the average predicts differences among the criterion 
activities. This second strategy maximizes multiple differential predic- 
tion (Horst, 195^)’ 

The two strategies differ not oniy by selecting different predictors 
but in the appropriateness of these selections for two contrasting de- 
cision situations. It is reasonable to expect the absolute prediction 
strategy to select as predictors variables highly related to whatever is 
coamon among the criterion activities, and the differential strategy to 
search for measures correlated with differences between criterion activities. 



To adopt the distinction siiggested by Cronbach and Gleser (I965) absolute 
prediction would seem the better model for Institutional decision making 
and differential prediction more appropriate for individual decisions. 

^*^stitution 6 requirement most coomoniy is to choose individuals. 

Individuals are selected for assignment to a criterion activity if they 
are ejected to do better in that activity than other available Individuals. 
To accomplish this it is desirable to have the most accurate estimate for 
each individual of his success in each criterion activity. For individual 
decisions^ on the other hand, the choice is among criterion activities . 

It is more important for the individual to have accurate information about 
^diether he can e:qpect to do better in one activity compared with 
activity than to know with exactness how well he might do in any one of 
the activities. 

A characteristic comnon to both decision situations is the 
incomplct enes s of data relating to criterion activities. Where criterion 
activities are numerous it is frequently impossible to obtain performance 
all activities for any individual. For example^ in the univer* 
sity setting where course work is available in fifty distinct areas ^ the 
typ 5 .cal student in four years only samples eleven to fifteen areas. 

Because choice of criterion activities whether by the individual or the 
Institution is not independent of the individual's predictor data, the 
incon5>letoncss of criterion data clearly poses theoretical problems in the 
estimation of regression parameters. In practice, however, these problems 
are often ignored (Johnson, I959) . After all, the proof of the goodness 
of predicti(m is easily accomplished through cross-validation, at least 



as far as absolute prediction is concerned* One straightforwardly 
calculates correlations between predicted and actual criterion perform- 
ances for some subsequent sample and compares these correlations with the 
multiple correlations obtained vdien variables were selected. 

Unfortunately, this has also been the typical means of cross -validating 
differential predictions. Although Horst’s (195^) differential predictor 
selection technique is keyed to maximizing the prediction of differences 
between all pairs of criteria, his solution does not depend upon the 
availability of such criterion differences or even of correlations between 
criteria. Rather, predictor selection requires only "adequate” estimates 
of the intercorrelations of all the potential predictors and of the corre- 
lation of each of these measures with each criterion. 6i anting the 
adequacy of these estimates, differential predictor selection can thus 
proceed with quite incomplete criterion data. However, using these 
selected differential predictenrs and validating their use raises issues 
not faced with absolute prediction. The problem is that once variables 
are selected as differentisd predictors they tend to be used and ’Validated" 
as if they were absolute predictors. To predict the difference between 
two criteria the appropriate weighting for a predictor is simply the 
difference between the wel^ts that predictor would have if each of the 
two criteria were to be predicted separately. Thus, the best prediction 
in the least squares sense of the difference between achievement in 
sociology and biology, using a set of selected predictors, is obtained by 
weighting each of these predictors to obtain the best least squares pre- 
dictions separately of sociology and biology and then taking the difference 
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between these wei^tings. This simple relationship together with the 
greater ease of interpreting predictions for k criteria rather than 
[(k/2) X (k-1)) differences between criteria, leads advocates of differ- 
ential prediction to make the differential aspect of predictions more 
inqplicit than explicit, e.g., in the Washington Pre-College (WPC) Testing 
Program (WPC, 1969b). In this program success in each of many academic 
areas is predicted on ths basis of variables selected to differentiate 
among these areas and it is left to the user of these predictions, the 
high school student and his counselor, to make the comparisons pre- 

dictions and thus extract the differential information. The question of 
how best to present differential predictions certainly has not been 
adequately studied, but that question goes beyond the concern of the 
present paper. 

Ifowever, once differential predictors have been used not to provide 
direct predictions of differences but instead predictions of the individual 
criteria to be differentiated by the user , it is easy to see idiy validation 
of such predictions has rested on techniques developed for the absolute 
model. To cite from the WPC Testing Program again, weights are derived for 
the selected differential predictors to best estimate each of the 
"differentiable" criteria. These weights are then applied to predictor 
data for new sanqples and these prediction equations "validated" by corre- 
lating the resulting predicted achievements with obtained achievements 
(WPC, 1969a). Such a strategy validates the accuracy of prediction of the 
individual criteria but it does not answer the question of how well dif- 
ferences among criteria have been predicted. So far no attempt has been 
made to validate differential prediction in this latter sense. 




Method 



The present study demonstrates the direct validation of differential 
prediction for a limited sample of students entering the University of 
Washington (UW) as freshmen after having completed the WPC battery. As a 
part of a larger study 655 students were identified who entered UW fall 
1966 and took through spring I968 a minimum of two courses in each of 
five areas of study: freshman English^ college algebra, modem foreign 

languages^ natural sciences, and social sciences. Requiring that each 
student had an earned grade point average (GRA) in each of the five areas 
meant that differences in performance could actually be computed for all 
ten pairings of achievement areas. This group of Ss comprised 26^ of the 
tested fl^eshman class, the remainder of which did not have the above 
pattern of course work. 

The sample was split into halves by assigning alternate students from 
an ordering based on the serially assigned student admissions numiber to 
one of two groups. A complete intercorrelation matrix for the twenty WPC 
predictor variables identified in Table 1 , the five college GPA*s and ten 
GRA differences was calculated separately for the two groups. Group I 
(N = 528, 593 ^ female) correlations were utilized for predictor selection 
and Group II (K = 327 ^ ^ 1 ^ female) data for cross-validation. 

Two predictor selections were conducted on Group I correlational 
data. Six predictors were selected from the twenty to maximize absolute 
prediction for the five area GPA*s. Then, in the second selection, six 
predictors were selected to maximize differential prediction of these same 
five academic areas. In both instances Horst's sequential selection 
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strategies were enployed. For the selection of absolute predictors that 
measure was first selected which had the highest average correlation with 
the five criteria (Horst, 1955)* The second selected predictor was the 
one which, combined with the first, produced the highest average multiple 
correlation with the five criteria. Additional predictors were then 
selected one at a time in accordance with this rule. The selection of six 
differential predictors proceeded in the same fashion except that the 
function to be maximized at each stage was the average multiple correlation 
across all possible differences between the five GPA*s. According to 
Horst ( 195 ^) this was equivalent to maximizing the difference between the 
average variance of the five predicted GPA' s and the average of their 
covariances. 

Following each of these selections two predictor weightings based on 
Group I data were calculated. Weights were first derived for the members 
of each predictor set to maximize the efficiency with which each of the 
five college GPA*s could be estimated. Secondly, the two sets of selected 
predictors were weighted to predict each of the ten differences between 
these GPA's by subtracting the weight appropriate to one of the area GPA's 
from the weight for the other in the pair. For example, weights were 
derived to predict the difference, "English GPA minus Algebra GPA" by 
subtracting from the weights for predicting English GPA those for predicting 
Algebra GEA. Both sets of weights were then applied to the predictor data 
for Groups I and II and correlations calculated between predicted and 
actual GPA*s and GPA differences. 
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Table 1 

Order of Selection of Predictors 



Absolute : 


Predictors 




Rredictor 


A 


\ 


English Usage 


.688 


.688 


HS Nat Sci 6Bl 


.195 


.883 


Reading Cooj 


.153 


1.036 


Math Achieve 


.129 


1.164 


Sex 


.071 


1.236 


HS For Lang GPA 


.050 


1.286 



Differential Rredictors 



Predictor 


A 




Vocabulary 


.070 


.070 


Spelling 


.037 


.107 


RS Math GPA 


.034 


.141 


HS English GIA 


.038 


.179 


HS Nat Sci GPA 


VO 

CVI 

0 

• 


.205 


English Usage 


.022 


.227 



Note.— Variables unselected included HS Social Studies GPA^ HS Electives 
Age^ Reading Speedy Mechanical Reasoning^ Spatial Ability ^ Applied 
Mathematics j Ouantitative Judgement ^ Data Sufficiency ^ and Functional 
Relationships. 
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Results 

Table 1 indicates the predictors selected for Group I. Predictor 
variables are listed in the order in which they were selected. For the 
absolute predictor selections \ is Horst's (1955) index of absolute pre- 
diction efficiency, the sum of the multiple correlations between predictors 
selected and the five area GPA's. The A elements are the increments to X 
with each predictor selection. Similarly, for the differential predictor 
selections ^ is Horst's index of differential prediction efficiency 
(195^1^), proportional to the difference between the average variance and 
the average covariance of the area GIA's predicted at each stage. In- 
crements to this index are again tabled as A. The differential index, 
does not have the same interpretation as the index for absolute prediction. 
^ cap never exceed the difference between unity and the average inter- 
correlation of the area GIA's. In this study these intercorrelations were 
known and, for Group I, the resulting maximum was .1^57* Only two predic- 
tors, English Usage and HS Nat Sci GPA, were common to the two sets of 
selections. 

2 

Table 2 reports the squared multiple correlations, R , of the two 

selected sets of predictors with each of the five GPA areas for Group 1 

data. Also reported are squsores of correlations between predicted and 

2 

actual GIA's for Group II. These latter are tabled as r^J. The battery 
of absolute predictors had the higher index of absolute prediction 
efficiency, 1.29 as opposed to 1.08> but striking was the shrinkage of 
predictability associated with the last three GPA areas. On cross- 
validation the two sets of predictors provided essentially the same level 




of absolute prediction. 
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Table 2 

Multiple Correlations and Cross Validations 
for Five College Course Area GPA's 





Absolute 


Predictors 


Differential 


I^edictors 


Area GPA's 




2 

r ^ 
YI 




2 


English 


.2997 


.2117 


.31^^6 


.2554 


Algebra 


.175^ 


.0950 


.1195 


.0801 


Foreign Languages 


.2716 


.1719 


.2481 


.1658 


Natural Sciences 


.2602 


.1019 


.2012 


.1073 


Social Sciences 


.2795 


.1018 


.195^^ 


.1004 


Sum 


1.2862 


.6823 


1.0765 


.7070 


Average percent 
variance 


25.7 


13.6 


21.5 


14.1 



h 




Table 3 displays . results for the predictions of differences 
between the college area GPA's. Differential prediction efficiency, as 
indexed here by the sum of the squared multiple correlations, was only 
slightly better for the differential battery, .87 versus .85. This ad- 
vantage was, however, maintained in cross-validation. Also displayed in 
Table 3 are the correlations between the two GBl’s contributing to each 
difference. The size of these college area GPA correlations was not 
strongly related to the predictability of the corresponding differences 
but, interestingly enou^, the least correlated areas, E^lish Algebra 
did provide the most highly predictable GPA difference. 

Exami ning the cross-validation results, the absolute predictors 
accounted for about 13^ of criterion variability, avera g in g across the 
five GPA *8 (Table 2), vdiile the differential predictors accounted for only 
one-half that amount (6.5^) of the variance in the ten GIA differences 
(Table 3 ). Before discounting this differential prediction effort, it 
should be noted that reliability of GPA differences was y a 

limiting factor. Given the correlations between the area GIA’s and assum- 
ing that the reliabilities of these course area GPA’s were as M gh as .70 
or .80, then the GPA differences would only have had reliabilities of 
approximately .50. There was, then, considerably less reliable variance 
to be accounted for in GPA differences than in the sioqple GPA's. 

Discussion 

What are the implications for the development of differential 
prediction? The goal, identifying variables which can be used to ±BpTove 
individual decision making, is obviously an important one. Althou^ the 
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Table 3 

Multiple Correlations and Cross Validations 
for Ten College GPA Differences 





Absolute 


Predictors 


Differential 


Predictors 


GPA r 


Difference GEA*s 




2 

X ^ 

YY 




2 

X ^ 

YY 


(Group 1) 


English - Algebra 


.1229 


.0997 


.1348 


.1090 


*31 


English - For Lang 


.0894 


.0865 


.1277 


.0937 




English - Nat Science 


.1049 


.0616 


.1084 


.0748 


.40 


English • Soc Science 


.0682 


.0123 


.0342 


.0019 


.42 


Algebra - For Lang 


.0847 


.0808 


.0900 


.0851 


.38 


Algebra - Nat Science 


.0654 


.0305 


.0511 


.0273 


.45 


Algebra - Soc Science 


.0895 


.0490 


.0832 


.0850 


.34 


For Lang - Nat Science 


.0917 


.0936 


.1030 


.0848 


.52 


For Lemg - Soc Science 


.0872 


.0595 


.0753 


.0420 


.46 


Nat Sci • Soc Science 


.0471 


.0269 


.0586 


.0465 


►57 


Sum 


.8510 


.6002 


.8663 


.6501 




Average percent 
variance 


8.5 


6.0 


8.7 


6.5 





12 



differential predictor selection strategy led in this study to slightly 
better prediction of GPA differences, the overall accuracy was very limited. 
Can this acciiracy be in^roved? The answer is yes if three separate aspects 
of the differential prediction situation can be properly investigated. 

These are the composition of the predictor pool, the matching of the 
potential predictors with relevant differences among the criteria, and the 
selection of appropriate criteria. 

To date, differential prediction efforts have largely depended upon 
the use of predictors which have established validity in the absolute 
prediction sense. For example, all of the WPG predictors are appreciably 
correlated with success in most traditional academic course areas. There 
is reason to believe that good differential predictors may not be like 
these. A recent prediction study with this same battery suggested that 
augmenting these aptitude and achievement measures with biographical and 
interest variables considerably improved differential prediction 
(Lunneborg, I968). Although these new variables were selected as dif- 
ferential predictors, they were not picked when the goal was maximizing 
absolute prediction. Increased variety in the predictor pool, then, is 
one way in -which differential prediction can be improved. 

The variety of predictors, however, is not sufficient to establish 
their usefulness. Clearly, predictors must be related to what is different 
among criteria. In this study, that the most predictable difference was 
between English and Algebra GPA’s may be understandable. The WPG battery 
measures, largely, r?rbal and quantitative skills. These skills sbould be 
useful in distinguishing subsequent English and Algebra achievements. If 
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the difference between English and Social Sciences GE^'s is to be predicted^ 
variables must be found which are similarly differentially related to 
success in these areas. 

Finally ; and perhaps most in^ortantly^ differential prediction depends 
upon criteria being differentiable. If two criteria^ within the constraints 
of their respective reliabilities ^ measure the same things it is useless to 
attempt a search for measures which will predict the difference between 
them. There is evidence in this study ^ for exaoqple^ that the correlation 
between success in Natural Science and success in Social Science ^ may 

rival in magnitude the reliabilities of these success measures. If this is 
true; then the difference between Natural and Social Science GPA's is a 
measure with essentially no reliable cooEponent to its variance and is 
necessarily unpredictable. It is not unlikely that there are a number of 
liberal arts achievement areas which cannot be differentiated. The pros- 
pect; in the WFC program; of contrasting selected liberal arts achievement 
areas with vocational-technical training areas will undoubtedly provide 
criterion differences ^diich are sufficiently reliable as to make the search 
for varied and rationally related predictors worthwhile. 
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